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INTRODUCTION 

This article describes a participative, action evaluation process (Nouwens & Robinson, 
1991; Wadsworth, 1991) that was developed when Central Queensland University (CQU) in 
Australia was forced to replace existing university-wide Learning Management System 
(LMS) software used to support the development and delivery of online courses. A number 
of important lessons were learned during this process and this article offers a 
sociotechnical systems analysis (Owens, 1998) of this evaluation process to assist 
academic staff involved in evaluation and choice of similar enterprise systems to 
understand the complexities and implications of decisions they make. 

HISTORY 

In 1995, the CQU had 8357 effective full-time students, about half the students studying in 
the distance education (Central Queensland University, 2001). Previously there had been 
some computer based support for distance learners, particularly using email. From 1995 to 
1999, a small but increasing number of online courses began to be developed to 
supplement distance and face-to-face modes of study. Long experience with print-based 
distance education dating back to 1978, and the early, gradual introduction of online 
courses, supported a steady development of online infrastructure and a progressive 
development of the technical and online teaching skills. 

Growing interest in online teaching as a way to expand postgraduate programs into niche 
markets and to provide distance and on-campus students with additional resources and 
support made it difficult for the University's multimedia development centre to support the 
increasing demand for online courses. In 1999, the University began to investigate the 
adoption of a Learning Management System to enable teaching staff to develop and 
manage online courses with little professional support. The term Learning Management 
System (LMS) refers to an integrated set of networked, computerised tools that support 
online learning (Virtual Learning Environment or Course Management System are other 
terms that are sometimes used). Typically, these systems include tools like discussion lists 
and chat rooms for online communication, automated quizzes and other assessment tools, 
tools that make it easier to prepare content for online presentation and some tools to 
monitor and manage student activities. 

A survey and a simple, technical evaluation of learning management software available in 
1999 led to the choice of WebCT for major LMS trials. It had the functionality required, the 
cost was low, it was well supported, it was compatible with the University's IT 
infrastructure and it promised to allow more efficient production. Centralised funding and 
support led to a rapid up take and, by late 2002, 231 courses had been developed to 
provide online course resources and/or support. 

Then, early in 2002, the University was informed that its then current version of WebCT 
would not be supported after 2003. Either of the alternative versions of WebCT would 
involve some increase in cost and an allocation of resources to support the change. CQU 
was in a position where external forces (WebCT software changes) required a major, 
unplanned change and a major reallocation of resources to support the change. In contrast 
to the brief evaluation in 1999, in 2002 there was a much wider selection of sophisticated 
LMS products on the market and CQU had made a significant investment in the use of an 



LMS, both in monetary terms and in the time and commitment of academic and support 
staff in preparing online resources. Three years of experience had exposed shortcomings of 
the system being used, as well as the 'risk' attached to a commitment to the use of an 'off 
the shelf' LMS. However, there was still a growing demand for the benefits offered by an 
LMS in the development of online courses. 

This evaluation did not seek to make judgements about the effectiveness of an LMS as a 
means for improving the effectiveness of learning in higher education; it followed the 
approach of Dutton, Cheong and Park (2004), whose case study inquires into the "variety 
of technical, institutional, social and economic constraints on the innovation process" (p. 
69). However, this (CQU) study focuses not on the initial introduction of an institution-wide 
LMS and the major changes that entails, but on the question of whether and how to change 
from one major enterprise LMS to another. 

THE EVALUATION APPROACH AND ITS INTERPRETATION 

The authors participated in the LMS evaluation process as project manager and facilitator, 
and were responsible for formulating and planning the evaluation approach and for 
managing the evaluation project to the point of choosing the LMS. Traditionally evaluation 
in education has had a summative focus, seeking to assemble information on which to 
make a judgement about the worth of some activity, tool or program (Smith & Lovat, 2004; 
Worthen, Sanders & Fitzpatrick, 1997). Such judgements have traditionally been made by 
experts, people from outside the organisation who can look objectively at the situation. 
Such experts obtain legitimacy from the application of analytical-empirical methodologies; 
they have what Habermas (1996) calls a "technical interest" in the evaluation. Smith and 
Lovat (2004, p. 188) argue that for such expert evaluations "the meanings and 
interpretations constructed by the evaluators were not accurate representations of the 
perceptions of the participants.. ..There is no one reality of a situation". Thus the model for 
valid and reliable evaluation in educational contexts has moved from a purely technical 
interest in the use of independent, external experts who objectively gather, analyse and 
make recommendations towards a practical interest in which participants in the context 
being evaluated negotiate and interpret meaning, and are supported to participate in and 
control the evaluation process and its outcomes. 

The transition from an expert evaluation to a participative evaluation model is consistent 
with an understanding that the culture of an organisation is not coherent and unitary, and 
that large organisations like universities actually contain within them many sub-cultures 
that both support and conflict with one another (Luck, Jones, McConachie & Danaher, 
2004; Schein, 1996). In this instance, the use of a participative evaluation process gave a 
voice to the main groups (sub-cultures) which had a strong interest in the evaluation 
process. Four main groups were identified: academics, information technology staff, 
multimedia development specialists and managers/administrators. Owing to limited time 
and the intervention of the summer vacation period, students were not included in the 
evaluation but their interests were represented by the academic evaluation group. 

To illuminate the evaluation approach described above, this article uses a sociotechnical 
systems framework to inform the interpretation of this process. Sociotechnical systems 
theory provides a framework for interpreting and understanding the interactions that occur 
in organisations subject to technological change. Owens (1998) provides a useful model 
that invites an exploration of interactions between four major interacting activity 
subsystems within an educational organisation (humans, structures, tasks, technologies) 
on the one hand, and the goals of the organisation and the contingencies of the 
environment in which the organisation operates on the other. (See Figure 1.) 

Figure 1: The educational institution as a sociotechnical system 
(Adapted from Owens, 1998, p. 78) 
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Owens' model is supported by more recent studies of the use of LMSs and the concept of 
the social shaping of technology (Dutton et al., 2004) that highlight organisational, 
cultural, economic and other factors influencing the process of technological change. The 
six components of Owens' model can be used to interrogate the situations facing the four 
interest groups (academic, technical, development support and administration) that were 
given a voice in the LMS evaluation process, and the University as a whole. 

Owens presents the four organisational subsystems, plus organisational goals and 
environmental contingencies, as components that exist in a dynamic, organic equilibrium. 
If changes occur or are intended in one component of the model, we should be mindful of 
responses that change generates in other components. In other words, if a decision is 
made to change the technology the University uses (i.e. WebCT), one should consider the 
consequences for changes in the tasks performed in the organisation, the effect on people 
and their skills and interests, the organisational structures in which people operate and the 
effects on other technologies already in use. The impact of the change on the goals of the 
University (e.g. resource allocation or marketing) and the response of the external 
environment to such a change should also be considered (e.g. the ability to attract 
additional students or additional funds or enrolments from industry). 

METHOD 

As described earlier, the University faced a version-change WebCT in the short-term. This 
would have involved some installation work and systems conversion work for technicians 
but little work in conversion of content. However, WebCT indicated that the life of the new 
version could be limited and that an upgrade to a more expensive enterprise system (Vista) 
would be required. These prospective changes, the trial nature of the original commitment 
to WebCT, pressure from some Faculties to move to other LMSs, problems experienced with 
WebCT in the CQU context and the fact that staff had developed some experience with the 
operation of a LMS led to a feeling among current LMS users that the University should 
conduct a rigorous evaluation of LMSs including an assessment of the issues and costs 
involved in changing to another LMS. 

Timeframe 

During the initial stages of the evaluation, project managers identified a shortlist of four 
LMSs for detailed evaluation based on lists of essential criteria developed by the four 
interest groups. Initial work with representatives from these groups established project 
milestones that would provide technical staff with time to install and test the new system, 
provide time for a trial to develop and deliver a small number of courses to students and 
provide time to convert all existing online courses to the new LMS for delivery in 2004. 
Only two months could be allowed for the evaluation of shortlisted LMSs, with another 
month set aside for reporting and the final decision to be reached. 



Stakeholders in the Change Process 

While senior management of the University was informed early in 2002 that a decision had 
to be made about choosing an LMS for 2004, no priority for action was given until the issue 
was raised at a staff development forum in November of that year. Teaching staff indicated 
that they required at least 12 months notice of such a change to allow them to develop 
appropriate skills and resources for new courses. This illustrates the different perspectives 
and priorities that can arise between what Luck et al. (2004) call the management sub- 
culture and the academic sub-culture. 

Pressure for change came from the grass-roots users of the LMS. To harness the energy for 
change and to build commitment to the final decision, project managers decided to involve 
critical stakeholder user-groups in a participatory action evaluation process to conduct a 
summative evaluation of commercially available LMSs and to make a recommendation to 
senior management to guide the decision to choose the University LMS for 2004 and some 
time into the future. The project manager's task was to involve representatives of the four 
critical stakeholder groups (academic, technical, development support and management) 
in the evaluation and decision-making process at three levels: 

1. In the plenary group with representatives of all four groups. 

2. In each of the four groups as they conducted separate evaluations of their 
facet of the four shortlisted LMSs. 

3. In the final collaborative process in which the four groups would justify and 
develop a recommendation with active involvement of senior management 
who carried final responsibility for the selection of the LMS. 

To make best use of the available time, each interest group worked largely independently 
to develop its own evaluation criteria appropriate for its area of expertise, and implement 
suitable testing and evaluation processes for the four shortlisted LMSs. 

A team of computer administrators looked at technical issues such as the reliability and 
robustness of the software, technical support and the LMSs' compatibility with other CQU 
computer systems and industry technical standards. CQU's multimedia development unit 
investigated the potential of each LMS for specialised development, IMS standards and 
implications for conversion of a large number of courses currently existing in WebCT. The 
administration group investigated contract conditions, pricing, commercial viability of the 
vendor, planned version changes and related commercial matters. The fourth group 
consisted of teaching academics and focussed on a range of issues that related to teaching 
and learning. The timing of the evaluation made it unfeasible to include students as 
members of this working party, but the teaching staff undertook to represent this group. 

While the management, technical and development stakeholder sub-cultures consisted of 
relatively small numbers of people, the academic issues group, representing the interests 
of a large group of disparate users, had first to consult academic staff to determine what 
issues were seen as important and to promote ownership of the outcome. 

The effort required by teaching staff to learn to use a different LMS application was one 
concern which has since been identified as a factor in reducing academics' motivation to 
teach online (Dutton et al., 2004). The academic working party evaluation team identified 
this as a major issue that was particularly of concern to those who had spent considerable 
time developing skills in WebCT. A number of institutional changes that had occurred at 
CQU in the preceding years had left the staff change-weary (Luck et al., 2004) and there 
were concerns about whether staff would support yet another change. Thus ease of use 
was one of the key criteria that emerged from this group. 

Decision-Making Process 

Evaluations involved a range of methods including where appropriate tests and trials of the 
LMSs, surveys, consultations with other users and vendor presentations. Each group 
prepared a written report indicating the evaluation process and criteria used, the 
justification of the criteria, the ranking of each LMS and the justification of the ranking. All 
parties then met with senior management to discuss the issues and to debate the 



advantages and disadvantages of various options, including the option to continue with the 
new version of WebCT. The final decision, to change to a different LMS, Blackboard, was 
made late in March 2003. 

Since the decision, installation and trials have proceeded successfully, with Blackboard in 
full operation for the March-June 2004 teaching period. Formative evaluation processes, 
including research and user forums, development teams and staff development for 
academic staff, are supporting current and future developments. 

DISCUSSION OF REFLECTIONS ON THE PROJECT 

Critical reflection on practice is an important element in the action evaluation process 
(Wadsworth, 1991). Organisational lessons learned by the evaluation team from this 
participatory evaluation process are outlined in the following paragraphs. 

This project demonstrated how vulnerable universities are likely to become to external 
decisions made by an LMS vendor to change or withdraw its product This evaluation 
process involved a holistic evaluation of both LMS software and the costs and benefits of 
changing the LMS at CQU. In just three years, the University had developed a substantial 
investment with over 230 courses using WebCT; however online access remains optional 
for most of these courses which are offered face-to-face and in print-based distance mode. 
As institutions develop a greater dependence on online learning, they should assess the 
risks of becoming 'locked-in' to particular LMS software when the cost of change becomes 
too great. Owens' (1998) framework invites one to question how responsive a 
sociotechnical system can be to changes in the external environment if it becomes tied to 
LMS software and a vendor who does not develop the software in the direction the 
institution desires to go. Discussions and reflection on this issue in plenary sessions 
explored the notion of 'middle-ware', a university controlled interface between students 
and the learning management software that could continue to give an institution a 
consistent interface in spite of changes from one LMS to another. 

Difficulty in transferring content from one LMS to another owing to conflicting data types is 
another concern that can be addressed by linking the content through a separate 
mediating system (Chapman, 2003). However, this approach of layering technological 
systems is likely to increase the impact on the humans, the structures and tasks and the 
goals of a university. Accordingly, one would expect that the sub-cultures that exist in a 
traditional university (Luck et al., 2004) would resist the changes that would make such 
technological change effective. The challenge for change agents is how to engage these 
sub-cultures in the processes of change while protecting the core values of education. 

It was useful to recognise a university as a complex organisation comprised of different 
sub-cultures that can operate semi-autonomously in a static world, but that need active 
intervention to help them work together in a dynamic and changing environment 
This difference in perspectives between senior management and academic course 
developers about timelines for decision-making regarding the choice of an LMS illustrates 
the point made by Luck et al. (2004) that one's understanding of an organisation can be 
enhanced by considering it not as a coherent culture, but as a collection of groups with 
different sub-cultures in collaboration and in conflict. A comment by one of the technical 
group suggesting that we should seek to change people' behaviour because information 
technology systems are difficult to change reinforces this point. 

Because the pressure for change came from grass-roots users and those with interests in 
the future use of an LMS, it was recognised that the legitimacy of the evaluation and the 
acceptance of the final decision depended on open and transparent processes that involved 
all critical stakeholder groups. The critical stakeholders are those who control parts of the 
process of using the LMS. Cultural Historical Activity Theory (CHAT) (Hasan, Gould, Larkin 
& Vrazalic, 2001) suggests that organisations can be understood as communities of human 
activity that function effectively by the division of labour guided by formal and informal 
organisational rules. 



This notion of the division of labour within an organisation is echoed in Owens' (1998) role 
of the task in sociotechnical systems, but focuses attention on the interdependence of 
tasks, and the influence that organisational sub-cultures (Luck et al., 2004) have on the 
flow of organisational activity. In a simple traditional university and a static operating 
environment, there is time to deal with problems, change practices and manage from the 
top of the organisation to overcome the reluctance of sub-cultures to change. A complex 
organisation like CQU, with 13 campuses, multiple modes of study and a diverse range of 
domestic and international students, and under government pressure to change, requires a 
university where the critical groups have sub-cultures of collaboration in order to obtain 
the benefits of change rather than cultures of resistance. The participative, action- 
evaluation approach adopted in this project can be used to support such organisational 
development. 

Evaluation in Dynamic Settings Should Involve Key Participant Groups 
Batchelor and Maxwell (1987) propose "a process in which the 'practitioners' are included 
as evaluators, which features collaborative planning and data-gathering, self-reflections 
and responsiveness" (p. 70). This principle of stakeholder involvement was a high priority 
for the evaluation process, not only to ensure that all relevant issues were identified and 
taken into account, but also to pave the way for the successful implementation of the 
chosen system. Acceptance of and even commitment to change is enhanced when 
'“Ownership' of the evaluation is vested in the 'Practitioners'" (Batchelor & Maxwell, 1987, 
p. 70). 

Involvement of the critical interest groups in the evaluation and the decision-making 
process led to general acceptance of the decision to change and to undertake the work 
associated with the change. The process also helped the major stakeholder groups 
(academic, technical, development, administration) to understand better the perspectives 
of each of these sub-cultures within the University and the labour contributed by each 
'division' of the educational enterprise. There are plans to carry this participant action- 
evaluation process into formative evaluation of the development of online courses, to 
promote scholarly, evidence-based practice models that involve all stakeholders in 
conversation and common collaborative effort. 

Successful change is promoted by a three-way balance among sub-cultural group interests: 
an interest group to drive change, bottom-up participant action and top-down support for 
change 

This project was initiated by a particular interest group (academic) that drives the 
University to respond to a problem. This group then proceeded to enlist key members from 
other critical groups in a bottom-up process to achieve change. Implications of the problem 
were articulated to obtain the support of senior management, which accepted the 
proposed evaluation plan. 

Other projects that have not proceeded so well have lacked these three components: a 
driving group, the participation of key interest groups and senior management support. 
Projects driven from the top or driven by an interest group without participant involvement 
seem to have a long and difficult gestation and short life. Warne, AN, Linger and Pascoe 
(2003, p. 307) make this point by drawing attention to the organisational conflict of 
understanding that arises between the practitioner-team's experiential understanding of a 
task and the abstract understanding of senior management. Smith and Lovat (2004) 
maintain that participation evaluation is effective because only the participants in a 
context really understand the detail and because 'experts' more distant from an activity 
often fail to interpret the cultural complexity of organisational operations. This LMS 
evaluation project shows that a participative decision-making process can be effective and 
can be completed quickly. 

Academic staff tend to revert to the pedagogy they normally 
use in face-to-face or distance education 

While the LMS evaluation, the LMS choice and the conversion process have been largely 
successful; encouraging academics to make both efficient and pedagogically effective use 
of the web for teaching has proved a greater challenge. The University seems to 
understand the need to direct resources to the evaluation and conversion process and to 



the purchase of software, but seems to assume that academic staff will use the system 
well. Dutton et al. (2004) have found that, when teaching online, many academics will 
default to traditional patterns of teaching. Owens' (1998) framework makes it clear that 
changing technology alone will not produce effective change; he argues that the tasks 
academics do, the structures they operate under and the way they relate to other people in 
the system must all change in a coherent way to make effective use of the new technology. 
The idea that sub-cultures within the University are likely to resist change (Luck et al., 
2004) adds to the problem of change, and suggests that a participative search for solutions 
would be more likely to be effective. 

CONCLUSION 

The CQU experience shows that changing an LMS has major consequences for academic, 
administration, technical and multimedia development staff. Evaluation to choose a new 
LMS should draw on the experience of all these groups and provides an opportunity for 
these groups to understand better the perspectives of the other groups. Substantial 
university resources are absorbed by such a change, yet the need to change may be beyond 
the University's control. Institutions should monitor the health and intentions of the vendor 
of their LMS software to obtain warning of change and manage risks associated with a 
sudden change of LMS. 

Owens' (1998) sociotechnical systems model highlights the need to involve the critical 
organisational units in evaluation and decision-making. It also draws attention to the 
impact that technology changes will continue to have on teaching tasks, university 
structures, participants and the goals of the university as LMS software and the web 
continue to develop. In this dynamic technological environment, the choice of technology is 
never a 'final' decision leading to a separate implementation phase. New problems can be 
expected to arise as the different interest groups learn to use new and more complex LMSs 
and identify strengths and weaknesses that were not apparent in the initial evaluation or 
have been introduced in upgrades. At CQU, regular user forums involving all the participant 
groups are conducted in a participative, formative evaluation process to address ongoing 
software and organisational change and to improve the LMS. 
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